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DETAILED ACTION 
Response to Amendment 

1. This office action is in response to communications filed 06-12-06. Claims 1,17, 
21, 22, 32, 33, 35, 46, 50 are amended. Claims 2-9, 13, 14, 18-20, 23-30, 34, 36-43, 
47-49, 51, 52 are original. Claims 54-60 are new. Claims 10-12, 15, 16, 31, 44, 45 are 
cancelled. 

2. The applicant did not make any references to the IDS objection. The objection is 
maintained. 

Response to Arguments 

3. Applicant's arguments filed 06-12-06 have been fully considered but they are not 
persuasive. See below for arguments. 

4. The applicant argues that Shoff does not disclose generating a single mask for 
each of the plurality of video frames and the masks include pixels and the pixels map to 
an indicia for identifying the region or video object" (Page 13) and that Shoff includes 
supplemental content which is hypertext document and it does not include any graphics 
data. The examiner respectfully points out that Shoff clearly discloses that the 
supplemental content includes graphics data (Column 5, lines 12-33) and if there is 
merchandise that is being sold or trivia question being asked then obviously the 
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graphics data is for a plurality of objects in a corresponding frame (Figures 8a-8c). 
Shoff clearly meets most of the claim limitations. Please see new rejections. 

5. The applicant argues that Srinivasan discloses "an authoring station which tracks 
an image and provides an annotation data stream" and a value table is generated for 
the tracked image and the value table does not contain any graphics data" (Page 14- 
15). Srinivasan clearly disclose that masks include graphics data (Column 6, lines 8- 
19). Srinivasan clearly meets most of the claim limitations. Please see new rejections. 

The applicant also argues that Srinivasan discloses a plurality of authoring 
stations and the tables are generated on an image-by-image basis instead of frame-by- 
frame basis (Page 15). Srinivasan discloses a mask or annotation for a video frame 
(Column 6, lines 8-19, Column 10, lines 47-49). The claims do not have any limitations, 
which disclose that the various annotation streams from authoring stations rather the 
claims disclose annotation or annotation data. 

6. The applicant argues that Wistendahl transmits N data for different frames, the N 
data only contains location coordinates, not actual graphics data associated wit the 
plurality of video objects (Pages 15-16). Wistendahl discloses that there are hot spots 
and graphics data associated with the authored link (Column 9, lines 58-67). The video 
source is transmitting frames to the subscriber terminal and the N data is a part of the 
IDM program with the graphics images and objects (Column 10, lines 8-62). The N data 
is the location coordinate of the hot spots or annotation data and mask of the frames 
(Column 6, lines 17-36). It is necessarily included that all screens have pixels and there 
are location coordinates for the pixels. 
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Information Disclosure Statement 

7. The information disclosure statement filed 1-1 1-02, 5-13-04 fails to comply with 
37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; 
each non-patent literature publication or that portion which caused it to be listed; and all 
other information or that portion which caused it to be listed. It has been placed in the 
application file, but the information referred to therein has not been considered. 

Claim Objections 

8. Claim 50 is objected to because of the following informalities: Claim 50 depends 
from cancelled claim 44. Claim 50 is assumed to depend from Claim 35. Appropriate 
correction is required. 

Claim Rejections - 35 USC § 103 

9. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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10. Claims 1, 4-6, 13-21, 32-35, 38-40, 46-60 are rejected under 35 U.S.C. 103(a) as 
being unpatentable over Shoff et al (US 6,240,555 and hereafter referred to as "Shoff ') 
in view of Wistendahl et al (US 6,496,981 and hereafter referred to as "Wistendahl") 
Regarding Claim 1, Shoff discloses a hyperlinked broadcast system (Figure 4, 
60), a continuous media server or video source (Figure 2, 42, Figure 4, 42) provides 
video information or programs (Figure 2, 40, Figure 4, 40) for a video program including 
a plurality of video frames (Column 10, lines 7-17) or another source such as satellite 
feed or another cable system (Column 5, lines 1-3), an enhanced content server or 
annotation system (Figure 2, 52, Figure 4, 52) generating supplemental content or 
annotation data with the video information or programs (Column 5, lines 12-23, Column 
12, lines 39-43) and generating timing information for synchronization of the program 
(Column 9, lines 66-67, Column 10, lines 1-17, Figure 6, 176, Figure 7, 182, Column 12, 
lines 43-47), an author constructing a target resource with target specification including 
the program, the supplemental content, and synchronization data combining them into 
a signal to be transmitted to the user (Column 12, lines 39-67, Column 14, lines 31-41), 
which would necessarily include the author receiving annotation data or supplemental 
content, video information or program information, and timing data of the program to 
synchronize the supplemental content into a an augmented video transmission signal 
or an augmented video information transmission generator associates the program with 
the supplemental content using timing or synchronization information (Column 12, lines 
39-67, Column 14, lines 31-41). Shoff discloses that the annotation data includes 
generating a single mask for each of the plurality of video frames (Figure 8a-8c) and 
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each mask including a plurality of pixels as it is necessarily included that each video 
frame has a plurality of pixels and a mask data will include a plurality of pixels, the 
plurality of pixels is mapped to an indicia for identifying a region or video object 
appearing in the corresponding video frame (Figure 8a, 204, Column 9, lines 41-59, 
Column 10, lines 44-58), each mask further including graphics data for at least one of 
the plurality of video objects appearing in the corresponding video frame such as 
merchandise (Column 5, lines 16-18, Column 9, lines 41-59, Column 11, lines 25-45). 
Shoff is silent on wherein each pixel associated with the same region or video object is 
mapped to the same indicia. Wistendahl discloses a system that has a movie or video 
program (Figure 1, 10), the movie or video has a plurality of frames (Column 12, lines 
1-2) and a frame with an object, the object can be clicked on to provide further 
information via a hot spot or mask (Column 6, lines 17-26). Wistendahl discloses that 
the annotation system generates annotation data includes generating a single mask for 
each of the plurality of frames (Column 10, lines 8-35, Figure 5a), each mask including 
a plurality of pixels (Column 10, lines 8-35), wherein each of the plurality of pixels is 
mapped to an indicia for identifying a region or video object appearing in the 
corresponding video frames (Column 10, lines 8-35), wherein each pixel associated with 
the same region or video object is mapped to the same indicia (Column 10, lines 8-35, 
Figure 2, Figure 5A, 50a-c), each mask further including graphics data for at least one 
of a plurality of video objects appearing the corresponding video frame (Column 10, 
lines 8-35, 57-61, Figure 2, Figure 5A, 50a-c). Therefore, it would have been obvious 
at the time the invention was made to one of ordinary skill in the art to modify Shoff to 
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include each pixel associated with the same region or video object is mapped to the 
same indicia (Column 10, lines 8-35, 57-61, Figure 2, Figure 5A, 50a-c) as taught by 
Wistendahl in order to use media content for interactive television (Column 1, lines 12- 
14, 45-67) as disclosed by Wistendahl. 

Regarding Claim 32, Shoff discloses a hyperlinked reception system comprising: 
a receiver or viewing computing unit receiving a transmission signal encoded with 
annotation data or supplemental content program (Column 10, lines 7-9), the annotation 
data including a single mask for each of a plurality of video frames of a video program 
(Figure 8a-8c), each mask including a plurality of pixels as it is necessarily included that 
each video frame has a plurality of pixels and a mask data will include a plurality of 
pixels, the plurality of pixels is mapped to an indicia for identifying a region or video 
object appearing in the corresponding video frame (Figure 8a, 204, Column 9, lines 41- 
59, Column 10, lines 44-58), wherein each of the plurality of pixels is mapped to an 
indicia for identifying a region or video object appearing in the corresponding video 
frame (Figure 8a, 204, Column 9, lines 41-59, Column 10, lines 44-58), each mask data 
packet including graphics data for at least one of plurality of video objects appearing in 
the video frame (Column 5, lines 16-18, Column 9, lines 41-59, Column 11, lines 25-45). 
Shoff discloses a video decoder or processor (Figure 1 , 92) extracting the annotation 
data from the transmission signal or the extracting timing information and presentation 
format information (Figure 6, Figure 7, 178, and a display device (Figure 1), a memory 
storing the EPG data (Figure 5, 104) and a central processing unit in communication 
with the memory the central processing unit being configured to synchronize the masks 
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or supplemental content with the video information on a frame by frame basis or frame 
numbers in response to the timing information or requirements (Column 10, lines 7-17, 
Figure 6, 176). Shoff is silent on wherein each pixel associated with the same region or 
video object is mapped to the same indicia and the memory which stores extracted 
annotated data. Wistendahl discloses a system that has a movie or video program 
(Figure 1, 10), the movie or video has a plurality of frames (Column 12, lines 1-2) and a 
frame with an object, the object can be clicked on to provide further information via a hot 
spot or mask (Column 6, lines 17-26). Wistendahl discloses that the annotation system 
generates annotation data includes generating a single mask for each of the plurality of 
frames (Column 10, lines 8-35, Figure 5a), each mask including a plurality of pixels 
(Column 10, lines 8-35), wherein each of the plurality of pixels is mapped to an indicia 
for identifying a region or video object appearing in the corresponding video frames 
(Column 10, lines 8-35), wherein each pixel associated with the same region or video 
object is mapped to the same indicia (Column 10, lines 8-35, 57-61, Figure 2, Figure 
5A, 50a-c), each mask further including graphics data for at least one of a plurality of 
video objects appearing the corresponding video frame (Column 10, lines 8-35, Figure 
2, Figure 5A, 50a-c). Wistendahl disclose that h the IDM program data can be stored 
together wit the N Data in association with the media content and transmitted together 
by the server to the subscriber's terminal (Column 7, lines 65-67, Column 8, lines 1-8) 
and that the console processor processes the IDM program data and N data and 
operates to the store the N data in the memory (Column 8, lines 40-45). Therefore, it 
would have been obvious at the time the invention was made to one of ordinary skill in 
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the art to modify Shoff to include each pixel associated with the same region or video 
object is mapped to the same indicia (Column 10, lines 8-35, 57-61, Figure 2, Figure 
5A, 50a-c) and a memory storing annotated or N data (Column 8, lines 40-45) as taught 
by Wistendahl in order to use media content for interactive television (Column 1 , lines 
12-14, 45-67) as disclosed by Wistendahl. 

Regarding Claim 35, Shoff discloses a method of generating a hyperlinked video 
signal (Figure 2, Figure 4) comprising generating annotation data or supplemental 
content time information from video information or timing requirements for supplemental 
content for a movie or program (Column 12, lines 39-47) for a video program including a 
plurality of frames (Column 10, lines 7-17); generating annotation data or supplement 
content for the video information or program (Column 12, lines 41-43), wherein the 
annotation data includes generating a single mask for each of the plurality of video 
frames (Figure 8a-8c) and each mask including a plurality of pixels as it is necessarily 
included that each video frame has a plurality of pixels and a mask data will include a 
plurality of pixels, the plurality of pixels is mapped to an indicia for identifying a region or 
video object appearing in the corresponding video frame (Figure 8a, 204, Column 9, 
lines 41-59, Column 10, lines 44-58), each mask further including graphics data for at 
least one of the plurality of video objects appearing in the corresponding video frame 
such as merchandise (Column 5, lines 16-18, Column 9, lines 41-59, Column 11, lines 
25-45). Shoff discloses communicating the timing information or requirements, the 
supplemental content and the program in order to create a signal by an author, which 
would necessarily include an augmented video information transmission generator as 
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the signal is being combined to be sent (Figure 2, Figure 4) and synchronizing the 
program with the supplemental content in response to the timing requirements from the 
author (Column 12, lines 39-47). 

Shoff is silent on wherein each pixel associated with the same region or video 
object is mapped to the same indicia. Wistendahl discloses a system that has a movie 
or video program (Figure 1 , 10), the movie or video has a plurality of frames (Column 
12, lines 1-2) and a frame with an object, the object can be clicked on to provide further 
information via a hot spot or mask (Column 6, lines 17-26). Wistendahl discloses that 
the annotation system generates annotation data includes generating a single mask for 
each of the plurality of frames (Column 10, lines 8-35, Figure 5a), each mask including 
a plurality of pixels (Column 10, lines 8-35), wherein each of the plurality of pixels is 
mapped to an indicia for identifying a region or video object appearing in the 
corresponding video frames (Column 10, lines 8-35), wherein each pixel associated with 
the same region or video object is mapped to the same indicia (Column 10, lines 8-35, 
57-61, Figure 2, Figure 5A, 50a-c), each mask further including graphics data for at 
least one of a plurality of video objects appearing the corresponding video frame 
(Column 10, lines 8-35, Figure 2, Figure 5A, 50a-c). Therefore, it would have been 
obvious at the time the invention was made to one of ordinary skill in the art to modify 
Shoff to include each pixel associated with the same region or video object is mapped 
to the same indicia (Column 10, lines 8-35, 57-61, Figure 2, Figure 5A, 50a-c) as taught 
by Wistendahl in order to use media content for interactive television (Column 1 , lines 
12-14, 45-67) as disclosed by Wistendahl. 



Application/Control Number: 09/943,583 Page 1 1 

Art Unit: 2623 

Regarding Claims 4, 34 and 38, Shoff and Wistendahl disclose all the limitations 
of Claims 1, 32 and 35 respectively. Shoff discloses that he timing information 
comprises at least one of frame number information (Column 10, lines 14-17). 

Regarding Claims 5 and 39, Shoff and Wistendahl disclose all the limitations of 
Claims 1 and 35 respectively. Shoff discloses the programs comprise digital video 
signal (Column 5, lines 4-5). 

Regarding Claims 6 and 40, Shoff and Wistendahl disclose all the limitations of 
Claims 1 and 35 respectively. Shoff discloses the programs comprise analog video 
signal (Column 5, lines 4-5). 

Regarding Claim 13, Shoff and Wistendahl disclose all the limitations of Claim 1. 
Shoff discloses a receiver or viewing computing unit in communication with a network or 
broadcast channel to the headend (Figure 2, 26, Figure 4, 68) which produces a signal 
with video information and supplemental content (Figure 6, Figure 7, 182); and a display 
device in communication with the viewing computing unit (Figure 2, 26, Figure 4, 66), 
and the receiver receives synchronized mask data or supplemental content with the 
video information on a frame by frame basis or frame numbers in response to the timing 
information or requirements (Column 10, lines 7-17, Figure 6, 176). 

Regarding Claim 14, Shoff and Wistendahl disclose all the limitations of Claim 
1 3. Shoff discloses that the display device displays the annotation data in response to a 
viewer request or the viewer has a displayed screen which allows the viewer to click on 
various buttons or URLs to see the time linked contextual information including playing 
games and details about actors (Column 11, lines 20-24, Figures 8a, 8b, 8c) 
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Regarding Claims 17 and 46, Shoff and Wistendahl disclose all the limitations of 
Claims 1 and 35 respectively. Shoff discloses that the mask data or supplemental 
content comprises location information of an object in an annotated video frame or 
location of the supplemental content based on the display format (Column 10, lines 44- 
50). 

Regarding Claims 18 and 47, Shoff and Wistendahl disclose all the limitations of 
claims 17 and 46 respectively. Wistendahl disclose the location information includes a 
graphics location reference that represents a fixed relation to a set of pixels associated 
wit the object (Column 6, lines 29-31). 

Regarding Claims 19 and 48, Shoff and Wistendahl disclose all the limitations of 
claims 18 and 47 respectively. Wistendahl discloses that there are a group of target 
pixels based on a tracking box and object (Figure 2, A). Therefore, it is necessarily 
included that the pixels include the upper left most pixel in the associated pixel set if the 
object that is being tracked is located at the upper most left corner (Figure 2, Figure 3). 

Regarding Claims 20 and 49, Shoff and Wistendahl disclose all the limitations of 
claims 18 and 48 respectively. Wistendahl discloses that there are a group of target 
pixels based on a tracking box and object (Figure 2 and Figure 3), if the object is in the 
center position then the centroid pixel is in the associated pixel set (Figure 2, 29). 
Therefore, it is necessarily included that the pixels include the centroid pixel in the 
associated pixel set. 

Regarding Claims 21 and 50, Shoff discloses all the limitations of Claims 15 and 
44 respectively. Shoff discloses that at least one of the mask data comprises location 
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information about an object in a video frame to be annotated or supplemental content to 
be added to video (Column 10, lines 44-47). Wistendahl discloses that location and 
shape information of an object in the video frame (Column 10, lines 8-35). 

Regarding Claim 33, Shoff and Wistendahl disclose all the limitations of Claim 
32. Shoff discloses that the receiver or viewing computing unit comprises a way to 
measure timer in the program to synchronize the supplemental content or masks to the 
associated program, which reads on a timer to calculate an offset of the timing 
information (Column 10, lines 7-15). 

Regarding Claim 51, Shoff and Wistendahl disclose all the limitations of Claim 
50. Wistendahl discloses the shape information is represented by a hyper link (Column 
10, lines 36-56). The hyperlink can be a graphical overlay of the object (Column 9, lines 
28-33). 

Regarding Claim 52, Shoff and Wistendahl disclose all the limitations of Claim 
50. Wistendahl discloses the shape information is represented by an outline of the 
object (Column 10, lines 25-28). 

Regarding Claim 53 and 57, Shoff and Wistendahl disclose all the limitations of 
Claims 1 and 32 respectively. Shoff discloses the marks include graphics data for a 
plurality of video objects appearing in the corresponding frame or merchandise, 
advertisements, information about the actors (Column 5, lines 18-32). 

Regarding Claim 54, Shoff and Wistendahl disclose all the limitations of Claim 1 . 
Shoff discloses each mask includes timing information of the corresponding video frame 
(Column 10, lines 15-17), wherein the augmented video information transmission 
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generator associates the video information with the annotation data (Column 12, lines 
44-47) based on a comparison of the annotation data timing information and the timing 
information included in the masks (Figure 6, 176, Column 10, lines 15-17). 

Regarding Claims 55 and 59, Shoff and Wistendahl disclose all the limitations of 
Claims 1 and 32 respectively. Wistendahl discloses on annotation data further includes 
a plurality of object data packets (Figure 3, 32, Column 17, lines 8-10), the mask further 
including an identifier or a hyperlink to an object mapping table or N data information 
(including values for several objects) (Column 6, lines 17-38, Column 7, lines 1-10) such 
as including at least one entry with an indicia from the corresponding mask identifying a 
particular video object (Figure 2, Column 6, lines 17-38), the entry further associating 
the indicia to information data structures included in one or more of the plurality of 
object data packets (Figure 2, Figure 3), the information data structures including 
information for particular video object (Column 9, lines 59-67, Column 10, lines 1-56). 

Regarding Claims 56 and 60, Shoff, Wistendahl disclose all the limitations of 
Claims 55 and 59 respectively. Shoff discloses the receiver being configured to overlay 
a graphics image on a particular video frame for the particular video object based on the 
graphics data included in the corresponding mask, based on the graphics data included 
in the corresponding mask (Figures 8a-8c). Wistendahl discloses the receiver being 
configured to overlay a graphics image on a particular video frame for the particular 
video object based on the graphics data included in the corresponding mask (Column 9, 
lines 59-67, Column 10, lines 1-56), retrieve the identifier of the object mapping table 
from the corresponding mask response to a user selection associated with the overlaid 
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graphics image (Column 6, lines 60-67, Column 7, lines 1-10, Column 9, lines 59-67, 
Column 10, lines 1-56); retrieve the object mapping table based on the retrieved 
identifier (Figure 3, Figure 7a); identify the indicia in the corresponding mask for the 
particular video object for which the graphics image was overlaid (Column 15, lines 3-5); 
locate the entry in the object mapping table with the identified indicia (Figure 5b, 51b, 
Figure 7a, Figure 7B); identify the information data structures associated with the 
located entry (Figure 5a, Figure 5b; retrieve the information in the identified information 
data structures; and display the retrieved information on the display device (Figure 7a). 

Regarding Claim 58, Shoff and Wistendahl disclose all the limitations of Claim 1. 
Shoff discloses each mask further includes timing information of the corresponding 
video frame, wherein the central processing unit (Figure 5, 92) synchronizes the 
plurality of masks with the video information based on the timing information included in 
the masks (Figure 6, 176). - 

1 1 . Claims 22-26, 29-30 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Srinivasan et al (US 6,357042 and hereafter referred to as "Srinivasan") in view of 
Wistendahl. 

Regarding Claim 22, Srinivasan discloses a hyperlinked transmission assembly 
system (Figure 1, Figure 8, Figure 10): a video information source providing video 
information or the video head end providing video data (Figure 1, 12, 15) for a video 
program including a plurality of video frames (Column 8, lines 13-14, Figure 4), an 
annotation data stream generator capable of accessing annotation data (Figure 1,11) 



Application/Control Number: 09/943,583 Page 16 

Art Unit: 2623 

including a single mask for each of the plurality of video frames (Column 6, lines 1-20), 
each mask including a plurality of pixels (Figures 2-4), wherein each of the plurality of 
pixels is mapped to an indicia for identifying a region or video object appearing the 
corresponding video frame wherein each pixel associated with the same region or video 
object (Figures 2-4), each mask further including graphics data for at least one of the 
plurality of video objects appearing in the corresponding video frame (Column 6, lines 1- 
20), , a frame reader/counter module or annotation data timing information decoder 
(Figure 10, 97) which is in communication with the video source as it is receiving video 
data and annotation data stream generator as it is receives the associated annotation 
stream, the frame reader/counter module adapts the annotation stream and video 
stream by counting the frames of the video data to associate the frames of a video 
stream with the annotation stream, which reads on extracting the timing information 
from the video data (Column 18, lines 22-35), an augmented or combined video 
information transmission generator or writer/data inserter in communication with the 
annotation stream generator or authoring system and the video information source to 
synchronize the video with the annotation data based on the timing information (Column 
18, lines 22-35, 48-60, Column 19, lines 48-60). Shoff is silent on wherein each pixel 
associated with the same region or video object is mapped to the same indicia. 
Wistendahl discloses a system that has a movie or video program (Figure 1 , 10), the 
movie or video has a plurality of frames (Column 12, lines 1-2) and a frame with an 
object, the object can be clicked on to provide further information via a hot spot or mask 
(Column 6, lines 17-26). Wistendahl discloses that the annotation system generates 
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annotation data includes generating a single mask for each of the plurality of frames 
(Column 10, lines 8-35, Figure 5a), each mask including a plurality of pixels (Column 
10, lines 8-35), wherein each of the plurality of pixels is mapped to an indicia for 
identifying a region or video object appearing in the corresponding video frames 
(Column 10, lines 8-35), wherein each pixel associated with the same region or video 
object is mapped to the same indicia (Column 10, lines 8-35, 57-61, Figure 2, Figure 
5A, 50a-c), each mask further including graphics data for at least one of a plurality of 
video objects appearing the corresponding video frame (Column 10, lines 8-35, Figure 
2, Figure 5A, 50a-c). Therefore, it would have been obvious at the time the invention 
was made to one of ordinary skill in the art to modify Shoff to include each pixel 
associated with the same region or video object is mapped to the same indicia (Column 
10, lines 8-35, 57-61, Figure 2, Figure 5A, 50a-c) as taught by Wistendahl in order to 
use media content for interactive television (Column 1, lines 12-14, 45-67) as disclosed 
by Wistendahl. 

Regarding Claim 23, Srinivasan and Wistendahl disclose all the limitations of 
Claim 22. Srinivasan discloses that the frame reader/counter module synchronizes the 
video data with the annotation stream on a frame-by-frame basis (Column 18, lines 22- 
35, 48-60, Column 19, lines 48-60). 

Regarding Claim 24, Srinivasan and Wistendahl disclose all the limitations of 
Claim 22. Srinivasan discloses that the annotation data timing information decoder or 
the frame reader/counter module inserter counts the frames in the video stream for 
synchronization and then inserts the stream denoting the frames in to the video blanking 
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interval or vertical blanking interval (VBI) so that the frame reader/counter module reads 
on a VBI decoder (Column 18, lines 48-60). 

Regarding Claim 25, Srinivasan and Wistendahl disclose all the limitations of 
Claim 22. Srinivasan discloses that the annotation data timing information decoder or 
the frame reader/counter module inserter counts the frames in the video stream for 
synchronization and then inserts the stream denoting the frames in to the video blanking 
interval or vertical blanking interval (VBI) so that the frame reader/counter module reads 
on a VBI decoder or a vertical ancillary data decoder (Column 18, lines 48-60). 

Regarding Claim 26, Srinivasan and Wistendahl disclose all the limitations of 
Claim 22. Srinivasan discloses that the timing information or frame number is counted 
by the frame reader/counter from the video data (Column 18, lines 22-35) and accesses 
the annotation stream provided by the authoring system as soon as the timing 
information is read or counted (Column 18, lines 22-35). 

Regarding Claim 29, Srinivasan and Wistendahl disclose all the limitations of 
Claim 22. Srinivasan discloses that the authoring system streams the annotation data 
that is associated with the video program based on the timing information which is used 
to synchronize the annotation stream to the video data (Column 18, lines 22-35, 48-60, 
Column 19, lines 48-60). 

Regarding Claim 30, Srinivasan and Wistendahl disclose all the limitations of 
Claim 22. Srinivasan discloses that the annotation data timing information comprises 
timestamp information (Column 28, lines 61-63), time code information (Column 18, 
lines 22-35), and frame number information (Column 18, lines 22-35). 
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12. Claims 2, 36, 41 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Shoff in view of Wistendahl as applied to claim 1 and 35 above, further in view of Hidary 
et al (US 5,774,664 and hereafter referred to as "Hidary"). 

Regarding Claims 2 and 36, Shoff and Wistendahl disclose all the limitations of 
Claims 1 and 35 respectively. Shoff is silent on the generator comprising a vertical 
blanking interval insertion device. Hidary discloses that a system providing videos 
(Figure 1, 4) and then augmenting the annotated data or supplemental content or URLs 
(Figure 1 , 8) to transmit to a viewer (Figure 1 , 12). Hidary discloses that the augmented 
video information transmission generator is a VBI insertion device (Column 4, lines 40- 
55). Therefore, it would have been obvious at the time the invention was made to one 
of ordinary skill in the art to modify Shoff in view of Wistendahl to include augmented 
video information transmission generator is a VBI insertion device (Column 4, lines 40- 
55) as taught by Hidary in order to provide a user friendly visual experience of television 
programming to a viewer (Column 1, lines 53-62) as disclosed by Hidary. 

Regarding Claim 41, Shoff, Wistendahl disclose all the limitations of Claim 35. 
Shoff is silent the insertion of timing information in the VBI. Hidary discloses that a 
system providing videos (Figure 1 , 4) and then augmenting the annotated data or 
supplemental content or URLs (Figure 1, 8) to transmit to a viewer (Figure 1, 12). 
Hidary discloses that the inserting URLs with timing information or time stamps in the 
VBI of an analog signal (Column 4, lines 36-37, 40-55). Therefore, it would have been 
obvious at the time the invention was made to one of ordinary skill in the art to modify 
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Shoff in view of Wistendahl to include inserting timing information or time stamps in the 
VBI of an analog signal (Column 4, lines 36, 37, 40-55) as taught by Hidary in order to 
provide a user friendly visual experience of television programming to a viewer (Column 
1 , lines 53-62) as disclosed by Hidary. 

13. Claims 3 and 37 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Shoff in view of Wistendahl as applied to claim 1 and 35 above, further in view of Eyer 
et al (US 5,982,445 and hereafter referred to as "Eyer"). 

Regarding Claims 3 and 37, Shoff and Wistendahl disclose all the limitations of 
Claims 1 and 35 respectively. Shoff discloses that the supplemental content and the 
video information can be a combined signal or two separate signals. Shoff is however 
silent on a multiplexer. Eyer discloses a system with programming services or video 
(Figure 1, 100) and annotation data or URL (Column 6, lines 18-24) to a multiplexer or a 
digital video data multiplexer (Figure 1,115, Column 7, lines 59-67, Column 8, lines 1- 
10). Therefore, it would have been obvious at the time the invention was made to one 
of ordinary skill in the art to modify Shoff in view of Wistendahl to include a multiplexer 
or a digital video data multiplexer to combine the annotated data to the video program 
(Figure 1,115, Column 7, lines 59-67, Column 8, lines 1-10) as taught by Eyer in order 
to provide viewers with additional information for educational and entertainment 
purposes (Column 4, lines 1-5) as disclosed by Eyer. 
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14. Claims 7-9 are rejected under 35 U.S.C. 103(a) as being unpatentable over Shoff 
in view of Wistendahl as applied to claim 1 above, further in view of Travaille et al (US 
6067107 and hereafter referred to as "Travaille"). 

Regarding Claims 7, Shoff discloses all the limitations of Claim 1 . Shoff 
discloses a post production environment or centralized headend (Figure 2, 22), a 
broadcast network (Column 4, lines 43-50), and the centralized headend combines 
video data and synchronized timing data and transmits to a headend or node (Column 
4, lines 43-50). Shoff discloses that the centralized headend has an author who 
synchronized the supplemental content to the program (Column 12, lines 39-47); the 
supplemental content and program can be a combined signal leaving the centralized 
headend or two separate signals (Column 10, lines 18-22). Shoff is silent on the 
regional node comprising the augmented video information transmission generator. 
Travaille discloses a broadcaster that provides program information (Figure 1, 114) and 
broadcast server provides interactive applications (Figure 1, 110, 112). Travaille 
discloses that the program and interactive applications are transmitted to a data 
insertion unit or node in order to combine the two signals via insertion into the VBI of the 
programming signal (Figure 1, 117) to be transmitted to the viewer (Figure 1, 120), 
which reads on the headend comprising the augmented video information transmission 
generator. Therefore, it wouid have been obvious at the time the invention was made to 
one of ordinary skill in the art to modify Shoff in view of Wistendahl to include that the 
program and interactive applications are transmitted to a data insertion unit or node in 
order to combine the two signals or the headend the augmented video information 
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transmission generator (Figure 1 , 1 17) as taught by Travaille in order to manage the 
interactive responses received by the interactive programs (Column 1, lines 28-31) as 
disclosed by Travaille. 

Regarding Claims 8, Shoff and Travaille disclose all the limitations of Claim 7. 
Shoff discloses that the node is a cable headend (Column 4, lines 45-52). 

Regarding Claims 9, Shoff and Travaille disclose all the limitations of Claim 7. 
Shoff discloses that the node is a satellite headend (Column 4, lines 45-52). 

15. Claim 27 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Srinivasan in view of Wistendahl as applied to claim 22 above, further in view of Shoff. 

Regarding Claim 27, Srinivasan and Wistendahl disclose all the limitations of 
Claim 22. Srinivasan is silent on the storage device. Shoff discloses a system, which 
has a video source providing video program (Figure 2, 40, Figure 4, 40) and an 
annotation data stream generator or enhanced content server (Figure 2, 52, Figure 4, 
52), which are augmented or combined to a single signal due to synchronizing the 
supplemental content to the program (Figure 4, 74). Shoff discloses that the annotation 
data stream generator accesses the annotation data from an internal storage device 
(Figure 2, 54, Figure 4, 54). Therefore, it would have been obvious at the time the 
invention was made to one of ordinary skill in the art to modify Srinivasan in view of 
Wistendahl to include the annotation data stream generator accesses the annotation 
data from an internal storage device (Figure 2, 54, Figure 4, 54) as taught by Shoff in 
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order to enable viewer interactively with video program (Column 1, lines 8-14) as 
disclosed by Shoff. 

16. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Srinivasan in view of Wistendahl as applied to claim 22 above, further in view of 
Purnaveja et al (US 6,006,241 and hereafter referred to as "Purna"). 

Regarding Claim 28, Srinivasan and Wistendahl discloses all the limitations of 
Claim 22. Srinivasan is silent on the external storage. Purna discloses a producer at a 
production station of annotation streams that defines the contents on display for the 
user (Figure 2, 210, 215) and web server provides annotation data (Figure 2, 230). 
Purna discloses that the production station or annotation data stream generator 
accesses annotation data from an external storage device or web server for 
textual/graphical inform such as a web page (Column 5, lines 43-53). Therefore, it 
would have been obvious at the time the invention was made to one of ordinary skill in 
the art to modify Srinivasan in view of Wistendahl to include that annotation data stream 
generator accesses annotation data from an external storage device or web server for 
textual/graphical inform such as a web page (Column 5, lines 43-53) as taught by Purna 
in order to delivery annotated multimedia streams over a diverse network while using 
network resources efficiently (Column 1, lines 64-67, Column 2, lines 27-33) as 
disclosed by Purna. 
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17. Claim 42 is rejected under 35 U.S.C. 103(a) as being unpatentable over Shoff in 
view of Wistendahl as applied to claim 35 above, further in view of Oguro et al (US 
2001/0033739 and hereafter referred to as "Oguro") and Hidary. 

Regarding Claim 42, Shoff discloses all the limitations of Claim 35. Shoff is silent 
the insertion of timing information in the vertical ancillary data or VBI. Oguro discloses a 
television broadcast system, which transmits digital video signals to the user (Page 5, 
paragraph 0077) and inserts data into the VBI of a digital video signal (Page 5, 
paragraph 0076). Hidary discloses that a system providing videos (Figure 1 , 4) and 
then augmenting the annotated data or supplemental content or URLs (Figure 1, 8) to 
transmit to a viewer (Figure 1 , 12). Hidary discloses that the inserting URLs with timing 
information or time stamps in the VBI of a digital video signal (Column 4, lines 36-37, 
40-55). Therefore, it would have been obvious at the time the invention was made to 
one of ordinary skill in the art to modify Shoff in view of Wistendahl to include inserting 
information in the VBI of a digital video signal (Page 5, paragraph 0076) as taught by 
Oguro in order to provide copy protect television broadcast programs (Page 1 , 
paragraphs 0001, 0006) as disclosed by Oguro. Therefore, it would have been obvious 
at the time the invention was made to one of ordinary skill in the art to modify Shoff in 
view of Wistendahl to include inserting timing information or time stamps in the VBI 
(Column 4, lines 36, 37, 40-55) as taught by Hidary in order to provide a user friendly 
visual experience of television programming to a viewer (Column 1, lines 53-62) as 
disclosed by Hidary. 
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18. Claim 43 is rejected under 35 U.S.C. 103(a) as being unpatentable over Shoff in 
view of Wistendahl as applied to claim 35 above, further in view of Purna. 

Regarding Claim 43, Shoff discloses all the limitations of Claim 35. Shoff is silent 
on the transmitting the timing information and video information to a broadcast network 
and subsequently to the augmented video transmission generator. Purna discloses a 
producer at a production station of annotation streams that defines the contents on 
display for the user (Figure 2, 210, 215) and web server provides annotation data 
(Figure 2, 230). Purna discloses that the production station or annotation data stream 
generator provides the timing information or time stamps (Column 7, lines 20-27) with 
the video information to the stream server or the augmented video information 
transmission generator (Column 7, lines 58-64). Therefore, it would have been obvious 
at the time the invention was made to one of ordinary skill in the art to modify Shoff in 
view of Wistendahl to include that production station or annotation data stream 
generator provides the timing information or time stamps (Column 7, lines 20-27) with 
the video information to the stream server or the augmented video information 
transmission generator (Column 7, lines 58-64) as taught by Purna in order to delivery 
annotated multimedia streams over a diverse network while using network resources 
efficiently (Column 1, lines 64-67, Column 2, lines 27-33) as disclosed by Purna. 

Conclusion 

19. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 
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§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Farzana E. Hossain whose telephone number is 571- 

272- 5943. The examiner can normally be reached on Monday to Friday 8:00 am to 
4:30 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Christopher Grant can be reached on 571-272-7294. The fax phone 
number for the organization where this application or proceeding is assigned is 571- 

273- 8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 
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